Method and system for video collection and analysis thereof

ABSTRACT

A computer implemented method of performing video surveillance on a computer system, wherein the computer system comprises a video and a display with a graphical user interface, the graphical user interface including a source window to view play back of the video, the surveillance method comprising: determining an area of interest of a frame of the video based upon a selection of the area of interest within the source window, and generating a plurality of windows within the graphical user interface, each of the windows displaying the area of interest of the frame of the video at a point in time, wherein the graphical user interface includes controls to manipulate the playback of the video from a point in time relative to each displayed portion.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application Ser.No. 61/040,468 filed Mar. 28, 2008, entitled “METHOD AND SYSTEM FORVIDEO COLLECTION AND ANALYSIS THEREOF,” the entire disclosure of whichis incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to a method and system for collecting andanalyzing data feeds from various sources.

BACKGROUND OF THE INVENTION

There are various fields, such as video surveillance, where multiplestreams of data, from various sources, may be collected and analyzed.For example, signals from one or more video cameras and/or othermonitoring sensors may be collected and simultaneously sent to a controlstation to be displayed on one or more display monitors. In addition tobeing displayed, the visual streams may also be recorded and stored.

In a control station, for example the monitors displaying the visualinformation from the video cameras, may be viewed and analyzed by ahuman operator. The human operator may be, for example, a securityofficer, who will be scrutinizing the displays for suspicious activity.This analysis may be performed on the live feeds coming in from thecameras, or it may be performed on earlier, stored signals.

In the past, the analysis process of this visual information may becomequite daunting, especially when surveillance is performed with manycameras and/or on a large area. Since a human operator has alllimitations native to humans, he or she may miss certain events that aredisplayed on the monitors. It is therefore advantageous, to provide asystem that can assist human operators analyzing large amounts of visualdata that solves the deficiency of the prior art systems.

In order to overcome the above deficiencies in the prior art, a systemis needed to provide rich camera management that enables sophisticatedrecording and archiving of video and automated video alerts based onintelligent analysis and detection. A system is needed which allowsusers monitoring multiple video and other monitoring sources to easilyaccess and interpret such information and data in a manner that allowsgreater flexibility and enhanced functionality.

SUMMARY OF THE INVENTION

In one embodiment there is disclosed a computer implemented method ofperforming video surveillance on a computer system, wherein the computersystem comprises a video and a display with a graphical user interface,the graphical user interface including a source window to view play backof the video. The surveillance method comprises the steps of determiningan area of interest of a frame of the video based upon a selection ofthe area of interest within the source window, and the step ofgenerating a plurality of windows within the graphical user interface,each of the windows displaying the area of interest of the frame of thevideo at a point in time, wherein the graphical user interface includescontrols to manipulate the playback of the video from a point in timerelative to each displayed portion.

In another embodiment, there is disclosed a computer implemented methodof detecting activity within a video on a computer system, wherein thecomputer system includes a video and a display with a graphical userinterface. The surveillance method comprises the steps of acquiring areference frame from the video, acquiring a comparison frame, comparingthe pixels of the reference frame to the pixels of the comparison frameto detect pixels that have changed, showing at least a portion of thevideo on the graphical user interface, showing the change in pixels onthe graphical user interface by coloring the changed pixels with apre-defined color, and gradually altering the different color to revealthe actual color of the changed pixels.

In another embodiment, there is disclosed a computer implemented methodof performing video surveillance on a computer system, wherein thecomputer system comprises a video and a display with a graphical userinterface, the graphical user interface including a map designating alocation of at least one monitoring sensor and a source window to viewplay back of a video. The surveillance method comprises the steps ofdisplaying a video being detected by a monitoring sensor and displayinga section of the map representing the location of the monitoring sensordetecting the video.

BRIEF DESCRIPTION OF THE FIGURES

Preferred features of the present invention are disclosed in theaccompanying drawings, wherein similar reference characters denotesimilar elements throughout the several views, and wherein:

FIG. 1 is an overview of system components in accordance with anembodiment of the present invention;

FIG. 2 is an overview of the hardware system architecture in accordancewith an embodiment of the present invention;

FIG. 3 is an overview of the software modules of a system in accordancewith an embodiment of the present invention;

FIG. 4A is a flow diagram of an exemplary operation of the monitoringsystem in accordance with an embodiment of the present invention;

FIG. 4B is a flow diagram of an exemplary operation of the eventhandling aspect of the system in accordance with an embodiment of thepresent invention;

FIG. 5 is an overview of hardware control modules of the system inaccordance with an embodiment of the present invention;

FIG. 6 is an overview of an embodiment of an audio mixer in accordancewith an embodiment of the present invention;

FIG. 7 is an overview of an audio stream handling process in accordancewith an embodiment of the present invention;

FIG. 8 is an overview of a display engine of an embodiment of thepresent invention;

FIG. 9 is an overview of a mapping graphical user interface inaccordance with an embodiment of the present invention;

FIG. 10 is an overview of a video review graphical user interface inaccordance with an embodiment of the present invention;

FIG. 11 is an overview of a video review graphical user interface inaccordance with an embodiment of the present invention;

FIG. 12 is an overview of a motion detection based search analysisinterface in accordance with an embodiment of the present invention;

FIG. 13 is a screenshot diagram of exemplary controls for framecomparison in accordance with an embodiment of the present invention;

FIG. 14 is an overview of the flow of the drill down module inaccordance with an embodiment of the present invention;

FIG. 15A is an overview of a thumbnail review graphical user interfacein accordance with an embodiment of the present invention;

FIG. 15B is a section of an overview of FIG. 15A showing the selectionof an area of interest;

FIG. 16 is an illustration of offset removal in accordance with anembodiment of the present invention;

FIG. 17 is an illustration of normalization in accordance with anembodiment of the present invention;

FIG. 18 is an illustration of aspect ratio adjustment in accordance withan embodiment of the present invention; and

FIG. 19 is an illustration of edge adjustment in accordance with anembodiment of the present invention.

DETAILED DESCRIPTION OF THE CERTAIN EMBODIMENTS

Certain embodiments of the present invention will be discussed withreference to the aforementioned figures, wherein like referencednumerals will refer to like components. It should be noted thatreferences in the specification to phrases such as “one embodiment” or“an embodiment” means that a particular feature, structure orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the invention. The appearances or phrasessuch as “in one embodiment” in various places in the specification arenot necessarily, but can be, referring to same embodiment. Theembodiments illustrate a system that integrates intelligent analyticsand enables sending camera views to local or remote displays over thenetwork. The embodiments further illustrate a system that greatlyenhances productivity by facilitating and expediting video review andanalysis by a human user.

Generally speaking, in one embodiment of the invention a monitoringsystem assists users in monitoring events, and includes a scalablecomputer system, operative with programming and related databases, thecomputer system connected to one or more monitoring sensors such as, forexample, video cameras, network video recorders, scanners, keycardsensors, infra-red cameras, microphones, access control systems,pressure monitoring systems, or other similar devices. The system allowsoperators to quickly monitor a large number of cameras and auxiliarymonitoring sensors in an efficient and expeditious manner.

In one embodiment a monitoring system comprises a plurality of variousmonitoring sensors deployed throughout an area that is subject toobservation. The monitoring system preferably receives information fromthese monitoring sensors or monitoring devices and presents theinformation to the users in a structured way that allows the users tobetter monitor the situation.

For example, the monitoring system may provide a user of the system witha visual interface showing locations of various monitoring sensorsoverlaid over a schematic representation (such as a map) of an areabeing monitored. Thus, a user may, for example, select to view a streamof video information from a specific camera based upon its location inthe area being monitored instead of having to keep a mental map of thevarious locations of the monitoring sensors. This may allow the user tofind the location of a specific event or follow the movement of asubject of observation from the field of view of one monitoring sensorto the field of view of another monitoring sensor, faster and moreefficiently than would be otherwise possible.

In various embodiments of the monitoring system, the system may mergevarious types of data, received from various monitoring sensors, inorder to present the user with an optimal overview of a situation. Sincevarious monitoring sensors provide various types of data, for examplevideo cameras provide video streams, microphones provide audio data,access control systems provide events such as notifications of incorrectpass key entries, or other types of data, it is useful to have thisinformation merged together in a manner that allows a user to extractthe necessary information in an expedient manner. For example, a userviewing a stream of video information received from a video camera, mayalso receive information from a keypad monitoring sensor that is locatedin the video camera's field of view. This will allow the user tovisually monitor the video feed data and monitor the data from thekeypad monitoring sensor simultaneously. In certain embodiments themonitoring system may also analyze streams received from variousmonitoring sensors and embed metadata into those streams. As describedin further detail below, information added to metadata may be anythingrelevant, for example a description of events being observed by themonitoring sensor, type of monitoring sensor performing the observation,time and date of observation, environmental properties at time ofobservation (e.g., temperature), or other related data.

In various embodiments of the monitoring system, the informationreceived from various monitoring sensors may be analyzed by the system,in order to provide the users of the system with alerts to facilitateeffective surveillance. For example, certain events or sequences ofevents may trigger corresponding alerts. If, for example, there is aloud sound in the vicinity of a particular camera that is sensed by, forexample, a microphone, an alert may be sent to a user and a windowdisplaying the stream of video information from that camera can bedisplayed to the user. Alerts and events can be configured to occur as aresult of any event or sequence of events. Various alerts and events aredescribed in further detail below.

In various embodiments of the monitoring system, the informationreceived from various monitoring sensors may be analyzed by the system,and enhancements may be added to the information. For example, a videostream received from a video camera may be analyzed for motion, andwhere motion is detected the images may be altered or enhanced, by, forexample, adding a red hue over the moving object. While any type ofanalysis and enhancement may be employed, certain types of analysis aredescribed in further detail below.

In various embodiments, the monitoring system can provide users withadditional functionality to assist them with reviewing streams of videoinformation. By way of non-limiting example, monitoring system may allowusers to analyze the stream of video information as a time line, byproviding a user with a control that allows the users to select and viewspecific time spans from the stream of video information, and tobookmark those time spans. These bookmarks may contain additionalinformation, such as, for example, information related to why thebookmark was placed, who should review the bookmark, or any otherrelevant information.

As described in further detail below with reference to FIGS. 15A and15B, monitoring system 100 can allow users to focus on specific areas ofstreams of video information by giving users the ability to pan to, tiltto, or zoom in on, a certain area of a video stream. The system can alsoinclude the functionality that permits users to simultaneously viewmultiple videos of differing resolution sizes, from various points intime, of the area the user focused on. It should be noted that themultiple videos can be thumbnail sized or can be configured to be anysize depending on the vision of the user, and the resolution, size, andnumber of monitors displaying the multiple videos.

Although the methods and systems are referenced primarily in the contextof surveillance and video cameras, it will be understood by persons ofordinary skill in the art that the present invention and its variousembodiments could be applied to any situation where monitoring ofstreams of information is useful. Moreover, it will be understood thatthe methods and systems described herein are not limited to anyparticular type of monitoring device.

With reference to FIG. 1, there is shown and described an overview of anembodiment of a monitoring system 100. As will be described in furtherdetail below, the system preferably includes system modules 110(described in detail with reference to FIG. 3) that enable functionalityof the monitoring system 100, hardware control modules 120 (described indetail with reference to FIGS. 5-8) that preferably enable the systemmodules 110 to communicate with the hardware components 130 ofmonitoring system 100, and hardware components 130 (described in detailwith reference to FIG. 2) which allow the system to operate and interactwith users.

Because different systems and hardware components frequently havedifferent syntaxes for information streams and the accompanying streamsof metadata related to those information streams, a proxy module 125 maybe included in certain embodiments of monitoring system 100, to performthe function of an interpreter thereby, for example, allowing thirdparty hardware, such as cameras, to be communicatively connected to thesystem. By way of non-limiting example, proxy module 125 may be used toconvert the metadata specific to a hardware or software component thatsends or creates the metadata to a syntax neutral format.

With reference to FIG. 2, there is shown and described an embodiment ofa monitoring system 100 including monitoring computers 215, 220 mappingstations 230, 235, monitors 245, databases (which may be part ofmonitoring computers 215, 220, mapping stations 230, 235, or separate),various monitoring sensors 210, and programming for providingfunctionality to one or more users. The components of monitoring system100 are preferably in communication with one another by beingcommunicatively interconnected via a network 225 which may be aninternal network, such as a local area network or LAN, or an externalnetwork such as the internet, extranet, wide area network, metropolitanarea network, or any other network. A monitoring sensor 210 can beconnected directly to the network 225 or a monitoring sensor 210 can beconnected to a video source 213, which can store the video informationstream from the monitoring sensor 210 as it sends the video informationstream to the network 225.

In various prior art monitoring systems, the various components would beconnected to hardware switching mechanisms that would include devicessuch as video matrix switches, quads, multiplexers, splitters, or othercostly and robust hardware devices. The use of these hardware switchesnecessitated large expenditures on purchase and maintenance of thoseswitches. Additionally, expansion of the hardware switching mechanismsand interconnection of various premises housing hardware switchingmechanisms is difficult, space consuming, costly, and limited. Further,the hardware switches would typically hinder the functioning of theprior art monitoring systems since signals sent by the users would bedelayed by having to go through the hardware switching mechanism beforereaching their destination. Furthermore, the use of hardware switchingmechanisms in surveillance systems necessitated long training foroperators and, in certain instances, the operators needed to memorizedetails about the hardware components in order to access them.

The use of communication over a network 225 using a protocol such asTransmission Control Protocol and Internet Protocol (“TCP/IP”) removesthe need for the hardware switching mechanism and expedites thefunctioning of the system. Use of Internet Protocol (“IP”) instead ofhardware switching, among other benefits, confers advantages such asbroad use of numerous devices all of which are compatible with themonitoring system 100, no vendor lock-in since IP switches can beobtained from a number of independent vendors; transmission errorresiliency since TCP allows reliable transmission of data through noisyenvironments; zero transmission loss since unlike analog copper basedtransmission that degrades based upon transmission distance, informationcan be sent using IP with no loss regardless of the distance; and easein set up and management as compared to the hardware switchingmechanisms.

Monitoring computers 215, 220 and mapping stations 230, 235 may be anytype of computer processor, for example, they may be personal computersystems running any type of operating system such as, for example,Microsoft Windows, Mac OS, or Linux. Monitoring computers 215, 220preferably include networking capabilities. Monitoring computers 215,220 may be configured for optimal use in monitoring applications andthus may include keyboards, touch-pads, joysticks, microphones,speakers, printers and/or any number of additional components andperipherals. In certain embodiments, the mapping stations 230, 235 caninclude some or all of the functionality of monitoring computers 215,220, and may even replace monitoring computers 215, 220.

The monitoring computers 215, 220 are preferably connected to at leastone monitor 245, though multiple monitors 245 may be utilized. Themonitors may be displays of any type including but not limited to LCD,projection, CRT, and any other type of display.

In use, the stream of information is sent by the monitoring sensor 210(which may be a video camera or any other monitoring sensor as describedabove), via network 225, to the mapping stations 230, 235. Mappingstations 230, 235, as described in further detail below, are preferablyused to analyze the stream and/or send the stream of information to aselected monitoring computer 215, 220 where the stream of informationmay be viewed and/or analyzed by the users of monitoring computer 215,220 on their respective monitors 245. It should be noted that themapping stations 230, 235 can be configured to send the stream ofinformation to every monitoring computer 215, 220, or select monitoringcomputers 215, 220 in response to input from users of mapping stations230, 235, according to an algorithm, or a combination of both.

With reference to FIG. 3, there is shown and described a diagram ofvarious system modules 110 of an exemplary embodiment of the monitoringsystem 100. The monitoring system 100 is just exemplary and should notbe interpreted as limiting the methods and systems, described herein, inany manner. Some of the modules of the monitoring system 100 aresoftware modules, while others are hardware components, as described infurther detail below. The system modules shown in FIG. 3 are softwaremodules that can be running on any number of computers or other hardwareelements of monitoring system 100 (e.g. monitoring computers 215, 220;mapping stations 230, 235).

As shown in FIG. 3, the software live video review module 360 representsa module that preferably controls monitoring sensors 110 and receivesstreams of monitoring information from those monitoring sensors 110 andpasses these streams to software mapping module 340, software eventcoordination module 320, and software video review module 380. Invarious embodiments the live video review module 360 may receiveinstructions and/or data from mapping module 340, event coordinationmodule 320, and video review module 380. By way of non-limiting example,the event coordination module 320 may, without user input, instruct livevideo review module 360 to instruct the monitoring sensor 210 to pan to,tilt to, or zoom in on an area where a pre-determined event is detectedas taking place. Additionally or alternatively, the mapping module 340may also instruct the live video module 360 to instruct the monitoringsensor 210 to pan to, tilt to, or zoom in on a particular area inresponse to a request by a user. As described in further detail below,live video review module 360 preferably allows users to conductmonitoring on the live information streams that are received

As described in further detail below, video review module 380 preferablyallows users to conduct monitoring on previously recorded informationstreams received from monitoring sensors 210. The information streamsmay be recorded by and received from video source 213, although inalternate embodiments the information streams may also be recorded bylive video review module 360.

Video review module 380 allows users to analyze the previously recordedstreams. As described in further detail below, the video analysis ofinformation streams may be assisted by analyzing specific time periodsof the video stream, using the software drill down module 390, andincorporating bookmarks into the recorded streams. Drill down module 390preferably divides the recorded streams of video information intosections of specific length of time, and displays those portionstogether on the monitor, to facilitate analysis. Bookmarks may be anytype of data or computer code that can be used by a software bookmarkdisplay module 370 to direct a user of a mapping station 230, 235 ormonitoring computer 215, 220 to a specific place or section of arecorded stream of video information. The bookmarks are preferablystored in an accompanying bookmark database controlled by the bookmarkdatabase module 330. The working of the bookmark display module 370 isfurther described below in reference to FIG. 11. The working of thedrill down module is described below with reference to FIGS. 14-19.

Mapping module 340 preferably provides the functionality of the mappingstations 230, 235. As described in further detail below, mapping module340 preferably manages (and allows users to manage) how and whichinformation streams (live streams from live video module 360 or recordedstreams video review module 380) are outputted to the users monitors245, via the remote video monitor module 310.

The event coordination module 320 detects events for which an alert oralarm may need to be issued. The stream of information sent bymonitoring sensor 210 can be received, via network 225, by eventcoordination module 320, which can analyze the stream of information todetermine whether a particular event is occurring before passing thestream to the event processing module 350. If an event is occurring(e.g. motion has been detected), event coordination module 320 may sendan alert or alarm to event processing module 350. Additionally oralternatively, event coordination module 320 may also embed information,related to the event being detected, into the stream received frommonitoring sensor 210, or into the metadata of that stream. By way ofnon-limiting example, the information embedded into the metadata or thestream may include information directed to which cameras, audio sources,and information streams are related to the alert.

As described herein, event coordination module 320 may detect certainevents by using data received from monitoring sensor 210 or by otherevent detection methods. For example, in certain embodiments a stream ofvideo information received from monitoring sensor 210 may be analyzed bythe event coordination module 320 to detect events such as loitering,motion, or a loud sound. In other embodiments, the monitoring sensor 210or the video source 213 may itself include its own software modules thatmay analyze the visual information it receives. For example, ifmonitoring sensor 210 is a video camera, the video camera itself maydetect events such as loud noises (if the video camera contains amicrophone), or motion, and embed information about those events intothe metadata accompanying the stream of information that video cameramonitoring sensor 210 sends via network 225 to the event processingmodule 350. Upon reception of a stream of video information from suchevent detecting video camera monitoring sensor 210, event coordinationmodule 320 may retrieve the information, about the events, from themetadata of the stream of video and use that data in its analysis.

When events are detected, software event coordination module 320 canfurther analyze the types and sequences of events. Event coordinationmodule 320 is preferably configured to issue alarms or alerts inresponse to specific sequences of events. By way of non-limitingexample, the event coordination module 320 may ignore an event if itdetects that multiple incorrect passwords were inputted by variouspeople into keypad monitoring sensor 210 in the field of view of a videocamera monitoring sensor 210, but may issue an alarm if a single personinputs multiple wrong passwords.

When event coordination module 320 detects an event or sequence ofevents that trigger an alarm or alert, the event processing modulepreferably sends the alert or alarm to the event processing module 350.Event processing module 350, upon receiving an alert or alarm, maytransmit instructions to the mapping module 340 and/or live video reviewmodule 360. By way of non-limiting example, the event processing modulemay send instruction to mapping module 340 to send a specificinformation stream to the remote video monitor module 310 which, inturn, can display the information stream (which may include a videoinformation stream, an alert or alarm, and/or any other relevantinformation) on monitors 245 of mapping stations 230, 235.

By way of another non-limiting example, the event processing module 320,may transmit instructions to the live video module 360, which mayinclude instructions relating to the operation of the monitoring sensors210 creating the information streams, for example, in the example above,describing a single person incorrectly inputting a password, the eventcoordination module 320 may instruct the monitoring sensor 210, whichcan be a camera, to pan to, tilt to, and zoom in on the face of theperson incorrectly entering the password. Event processing module 350preferably directs the alerts and/or alarms received from eventcoordination module 320 to the relevant monitoring computers 215, 220,mapping stations 230, 235 and their respective monitors 245. The alertsdisplayed on the monitors 245 may take on any form, for example, theymay be messages, sounds, flashing lights, or other warning methods asdescribed in further detail below. It should be noted that the eventprocessing module 350 may send alerts to every monitoring computer 215,220 or mapping station 230, 235 in the monitoring system 100, or to onlyselect monitoring computers 215. For example, alerts for major events(for example, an extremely loud sound such as an explosion) may be sentto every computer 215, 220, 230, 235 on the network, alerting the usersof a potential disaster, whereas alerts for minor events, for exampleloitering, may be sent to a specific individual's monitoring computer215 that is monitoring the field of view of a camera wherein theloitering is occurring. In certain situations the event coordinationmodule 320 may bypass the event processing module 350 and may, itself,send the alerts or alarms to the various monitoring computers 215, 220and or mapping computers 230, 235, to be displayed on physical monitors245 via remote video monitor module 310.

The alerts are preferably displayed to the users by the mapping module340 or live video review module 360 (depending on whether a user is in amapping mode or live video review mode) as described in further detailbelow with reference to FIGS. 9 and 10 respectively. As described indetail below, the users may then inspect the alert, by using GUI 1100 orGUI 1200, and mark the alert as handled and/or provide additionalinformation related to the alert. This input from the users ispreferably collected by the event processor module 350 and sent to eventcoordinator module 320 which may keep track of alerts and any operationperformed in relation to the alerts.

It should be noted that the various modules shown herein are forillustrative purposes only. In various embodiments the functionality ofthe modules may be assumed by a single software module, or severalsoftware modules. Likewise, the system should not be limited to anyspecific type of hardware, any type of computer, sensor, display device,or other components may be used.

With reference to FIG. 4A, there is shown and described an embodiment ofan exemplary operation of an embodiment of monitoring system 100. Itshould be noted that this exemplary operation should not be interpretedto limit the systems and methods described herein. In variousembodiments many other operations may be performed by and on themonitoring system 100. Additionally the system may perform the stepsdescribed below in a different order than described.

First, the monitoring system 100 receives a stream of information from amonitoring sensor 210 (e.g. camera) across network 225. Step 402.

Next, the monitoring system 100 analyzes the received information, asdescribed above, by for example using event coordination module 320.Step 404.

If an event is detected, the event coordination module 320 sends analert or alarm to a computer monitoring the area, for example monitoringcomputer 215. Step 406. As described above, this alert can be sent toone of a number of monitoring computers 215, 220 and mapping stations230, 235.

The monitoring system 100 may then add metadata to the stream. Asdescribed in further detail below, the metadata may include anyinformation relevant to the stream. The data handling may be performedby metadata stream handler 545, abstract metadata provider 560, specificmetadata provider 575, and metadata renderer 545. Step 408.

The monitoring system 100 then converts the received stream of videoinformation to a format understood by a monitoring computer 215. Theconversion may be performed, as described above, by the display engine515, video stream handler 540, abstract video provider 555, and specificvideo provider 570. Step 410.

The monitoring system 100 preferably sends the stream of videoinformation across network 225 to the monitoring computer 215. Step 412.

The monitoring system 100 preferably stores the stream of videoinformation somewhere on the monitoring system 100, for example a smallportion of the stream may be stored in cache, additionally oralternatively it can store the entire stream of video information forexample on video source 213 which may include a hard-drive, server,back-up tape, or any other storage medium (not shown) that is preferablyconnected to the monitoring system 100 via the network 225. Step 414.

If a user submits a proper input, as described below with reference toFIGS. 9-11, the monitoring system 100, by using layout module 520, maydisplay a window object containing the video stream to the user. Step416.

As described in more detail with reference to FIGS. 9-11, the monitoringsystem 100 preferably allows the user to perform operations such asplay, pause, rewind, on the video stream. Additionally, in response to aproper user input, the monitoring system 100 preferably allows the userto pan to, tilt to, or zoom in on a section of a video in a window. Step418.

As described herein with reference to FIGS. 15-19, in response to aproper user input, the monitoring system 100 preferably allows the userto display, on the monitor 245, thumbnail images captured at differentpoints in time from the video information monitor. Step 420.

With reference to FIG. 4B, there is shown and described anotherembodiment of an exemplary operation of another embodiment of monitoringsystem 100. It should be noted that this exemplary operation should notbe interpreted to limit the systems and methods described herein. Invarious embodiments many other operations may be performed by and on themonitoring system 100. Additionally the system may perform the stepsdescribed below in a different order than described.

First, the video source 213 of monitoring system 100 connects tomonitoring sensor 210 via network 225. Step 422. As discussed above inalternate embodiments the monitoring sensor 213 may be connecteddirectly to video source 213.

Next, the video information stream and its accompanying metadata, sentby monitoring sensor 210, can be received and recorded by video source213. Step 424.

Next, the video stream and its attached metadata is analyzed internallyby video source 213. Step 426.

If video source 213 detects an event, it preferably sends the eventdata, through proxy module 125, to event coordinator module 320. Step428.

Next, event coordinator module 320 can examine the event data anddetermine whether the event should cause an alert to be triggered. Step430.

Next, event coordinator module 320 can determine what monitoring sensors210 are relevant to the alert and can append this information to themetadata of the video information stream. Step 432.

Next, if a condition such as a specific level of importance of an alert,is ascertained, the event coordinator module 320 can send the videostream along with the alert to users of monitoring computers 215, 220 ormapping stations 230, 235. Step 434.

Alternatively, the alert may be sent to and received by event processingmodule 350, which in turn can send the alert to the mapping module 340or live video review module 360 to be displayed on a specific computeror station, for example monitoring computer 215, or multiple computersfor example mapping stations 230, 235. Step 436.

If the alert was forwarded to monitoring computer 215, the live videoreview module 360 can extract, automatically or in response to userinput, the relevant information from the metadata and, using the remotevideo monitor module 310, display the information stream and metadatafrom the relevant monitoring sensors 210 on monitor 245 attached tomonitoring computer 215. Step 438.

Monitoring sensors 210 display the information stream by firstconnecting to the video source 213 and requesting the live relay ofvideo from one or more monitoring sensors 210 to a predetermined monitor245. It should be noted that this is exemplary and in other embodimentsthe displaying of video from the monitoring sensors 210 on the physicalmonitors 245 may be performed in other ways, for example by having themonitoring sensor directly send the information stream to the monitor245, or in any other way heretofore known in the art or hereafterdeveloped.

Next, the operator of the monitoring computer 215, may view the videostream and receive information as to the cause of the alert. Step 440.

After reviewing the information the operator can take whatever action isnecessary to handle the situation (e.g. notify security that there is anunauthorized person loitering in a specified location). Next, theoperator of the monitoring computer 215, may acknowledge the alert andwhat steps have been taken. Step 442.

Next, the event processing module 350 can receive the acknowledgementand send the acknowledgement to event coordination module 320. Step 444.

Once the event coordination module 320 receives confirmation that thealert has been handled, the event coordination module 320 instructs allmapping stations 230, 235 and monitoring computers 215, 220 that thealert has been handled and no further action is necessary, or thatadditional action is necessary. Step 446.

This method allows for configurations of the monitoring system 100,where, for example, certain users can be responsible for specific alertsrelated to their expertise and/or duties. Additionally, all users of thesystem may be appraised of the status of various alerts and sectors ofthe area being monitored.

With reference to FIG. 5, there is shown and described variouscomponents of an embodiment of the hardware control modules 120 of amonitoring system 100, that perform processing and conversion of signalsreceived as information streams from monitoring sensor 210. In variousembodiments the shown software components can be loaded onto, and thehardware elements may be included as part of mapping stations 230, 235and/or monitoring computers 215, 220. As shown in FIG. 5, the systemincludes modules such as audio mixer 510, display engine 515, layoutmodule 520, audio stream handler 535, video stream handler 540, metadatastream handler 545, abstract audio provider 550, abstract video provider555, abstract metadata provider 560, specific audio provider 565,specific video provider 570, specific metadata provider 575, audiodecoder 580, and video decoder 585. These modules work together toenable functionality of the system. This list is meant to be exemplaryand in various embodiments more or fewer components may be utilized. Theinclusion of various components will depend on the particular embodimentof the monitoring system 100 and is a matter of design choice that willbe apparent to those skilled in the art, e.g. an audio mixer 510, may beexcluded from a monitoring computer 215 with no speakers.

Layout module 520 comprises a provider URL module 525 and a metadatarenderer 530. Layout module 520 may manage the positioning of windowobjects on the monitor and process/delegate user inputs such as, by wayof non-limiting example, mouse clicks, and key strokes. Layout module520 may also request audio, video and metadata stream handlers to updatethemselves. Layout module 520 is initialized by a text string thatdescribes the location of various visual elements. The layout stringsmay be provided in any way including by reading files on the hostcomputer, or receiving instructions from a central location, or througha network connection. Layout module 520 may be utilized to arrange thevarious visual components of the GUIs utilized by Users of themonitoring system 100.

Provider URL module 525 is used to locate a resource. By way ofnon-limiting example, such resource may be a video stream or an audiostream. A provider URL module 525 preferably receives the protocol,user-name, password and location of a certain resource. By way ofnon-limiting an example of a provider URL module 525 may take on theform of [protocol]://[user-name]:[password]@[host:port]/[providerspecific information]. It should be noted that any other method ofspecifying the location of a resource over a network, as is known in theart, may be utilized. The provider URL module 525 is preferably the waythat various components of the system locate resources across thenetwork 225. Metadata renderer 530 is used to extract metadata from theinformation stream received by the monitoring system 100, and outputand/or format the information contained within the metadata.

Audio stream handler 535, abstract audio provider 550, and specificaudio provider 565 are software modules that are used by the system tokeep track of, and direct audio streams. Whenever an audio stream isreceived by the monitoring system 100 via the monitoring sensor 210, orpreviously recorded stream via video source 213, the audio streamhandler 35, abstract audio provider 550, and specific audio provider 565are utilized to process the stream and deliver it to the monitoringcomputers 215, 220 or mapping stations 230, 235. The working of theaudio stream handler 535, abstract audio provider 550, and specificaudio provider 565 is described below with reference to FIG. 6. Thisallows the user of monitoring system 100 to listen to the sound producedby monitoring sensor 210.

Additionally, to process audio streams, and audio decoder module 580 ispreferably utilized. Audio decoder module is described below withreference to FIG. 7.

Video stream handler 540, abstract video provider 555, and specificvideo provider 570 are software modules that are used by the system tokeep track of, and direct video streams. Whenever a video stream isreceived by the monitoring system 100 via the monitoring sensor 210, orpreviously recorded stream via video source 213, video stream handler540, abstract video provider 555, and specific video provider 570 areutilized in order to process the stream and deliver it to the monitoringcomputers 215, 220 or mapping stations 230, 235. The working of thevideo stream handler 540, abstract video provider 555, and specificvideo provider 570 is described in further detail below.

Metadata stream handler 545, abstract metadata provider 560, andspecific metadata provider 575 are software modules that are used by thesystem to keep track of, and direct metadata embedded in informationstreams. Whenever an information stream is received by the monitoringsystem 100 via monitoring sensor 210, or previously recorded stream viavideo source 213, the metadata stream handler 545, abstract metadataprovider 560, and specific metadata provider 75 are utilized in order toprocess the stream and deliver it to the monitoring computers 215, 220or mapping stations 230, 235. The working of the metadata stream handler545, abstract metadata provider 560, and specific metadata provider 575is described in further detail below.

With reference to FIG. 6, there is shown an audio mixer 510 connected tothe audio stream handler 535, abstract audio provider, 550 and specificaudio provider 565 of the hardware control modules 120 of the monitoringsystem 100 according to an embodiment of the invention. The audio mixer510 preferably provides access to the audio hardware and handles mixingof multiple audio streams into one unified stream, which may be playedthrough an output device such as speakers. For example, the audio mixeris used to process the sound information received from a monitoringsensor 210 and to deliver the sound elements to the hardware sound card(not shown) of the monitoring computers 215, 220.

As seen in FIG. 6, in one embodiment, the components may include anaudio output module 610, an audio hardware driver 620, sound cardinterface module 630, audio decoder 580, abstract audio provider module550, and a specific audio provider module 565. In use, the audio decoderpreferably decodes h.711 or h.271 audio formats to their uncompressed8-bit 8 KHz raw format. The decoded bytes are then passed on to a bufferof the sound card interface module 630, which then performs the mixingof the audio with other audio sources. For example, audio informationfrom various monitoring sensors 210 may be combined into one stream. Itshould be noted that audio mixer 400 is a non-limiting example, any typeof buffers or decoders may be utilized to perform any type of relevantconversion to any type of format. The sound card interface module 630may be any type of software component such as for example MICROSOFTDirectSound.

With reference to FIG. 7, there is shown a diagram of audio streamhandling process of the monitoring system 100. In an embodiment of themonitoring system 100, in order to handle an audio stream the system mayutilize the specific audio provider 565, audio provider factory 730, andaudio provider url 525. The abstract audio provider 550 is anabstraction of an audio source containing the interface definitions offunctions such as connect, login, startlive, and any other relevantfunction. The specific audio provider 565 is an instance that containsthe implementation specific to the source. In use, audio providerfactory 730 preferably parses the protocol identifier and creates aspecific audio provider 565 that matches the specific source from whichthe audio stream is received. The specific audio provider 565 is thenmanipulated, by the audio stream handler 535 using the interface of theabstract audio provider 555.

After manipulation, the audio data is decoded by the audio streamdecoder 580 and passed to an output device (e.g. speakers, headphones,or any other sound output device) via audio mixer 510 as describedabove.

It should be noted that the above description of the audio processingcomponents of the monitoring system 100 are exemplary and may besubstituted with any other sound processing methodology. It will beapparent to those skilled in the art that various hardware and softwaremodules may be used to achieve the same goals of receiving,manipulating, and transferring sound streams. This allows the user ofmonitoring system 100 to listen to the sound produced by monitoringsensor 210.

With reference to FIG. 8, there is shown a display engine 515 of thehardware control modules 120 of the monitoring system 100 according toan embodiment of the invention. The display engine 515 preferablyprovides access to the video hardware and may enable outputting variousvideo streams to output devices such as monitors 245. For example, thedisplay engine 515 is preferably used to display a stream of videoinformation from a video camera monitoring sensor 210, as well asvarious GUI elements such as buttons, icons, windows, and other similarelements.

The display engine 515 may include a physical display driver 820,application programming interface 830, monitor class module 835, pixelshader module 845, texture cache 840, window module 850, and texturemodule 855. Display engine 515 preferably outputs to a physical display245.

The physical display 245 may correspond to any type of display asdescribed above. The display driver 820 enables the monitoring system100 to communicate with the physical display.

Application programming interface 830 may be any type of interface thatallows the monitoring system 100 to render graphics on the monitor. Byway of non-limiting example, the application programming interface maybe Direct X, Direct3D distributed by MICROSOFT Corp., or otherprogramming interfaces.

The monitor module 835 preferably corresponds to a video output on avideo adapter (not shown). The monitor module 835 handles the allocationof texture memory in the VRAM of the graphics adapter. The monitormodule 835 may be configured to utilize system memory if the graphicadapter runs out of memory.

Texture cache module 840 preferably corresponds to the cache of thevideo adaptor which may be used to hold frequently used textures andthus may help conserve VRAM.

The pixel shader module 845 contains instructions related to renderingof textures. The pixel shader module 845 may be used to instruct theGraphic Processing Unit of the video card how to convert various pixelformats supported by various video providers (by way of non-limitingexample, various surveillance cameras) to Red Green Blue Alpha (“RGBA”)format frequently used by graphic adapters, although it should be notedthat the invention should not be limited to any specific format. By wayof non-limiting example the pixel shader module 845 may be used toconvert YcrCb 4:1:1 or ABGR formats to RGBA.

The window module 850 is preferably a collection of instructions to drawvarious elements of related information within a region. The windowmodule may provide instructions related to the intricacies of dealingwith the monitor module 835. The window module 850 may further providevarious functions for drawing various geometric primitives such asgradients and filled rectangles, as well as the ability to draw texturessuch as rectangles. Window module 850 is also used to create windowobjects (described in further detail below) which are registered throughthe monitor module 835 and eventually displayed on the physical display245. The window objects created using window module 850 may repaintthemselves in each refresh cycle.

The texture module 855 contains various buffers and commands needed toload bitmap arrays (not shown) from memory, disk files, and/or resourcesembedded in executable image files. Texture module 855 may consist of aplurality of buffers that are combined through pixel shaders to create afinal RGBA output.

With reference back to FIGS. 5 and 8, in an embodiment of the monitoringsystem 100, in order to handle a video stream the system utilizes videoprovider 525, a video decoder 585, an abstract video provider 555, awindow module 850 and video stream handler 540. In use, the video streammechanism 700 may be initialized by passing a video stream URL to thevideo stream provider factory (not shown). Subsequently the video streamhandler 540 may receive instructions as to the resolution and pixelformat of the display window object 850 associated with the video streamhandler 540. The resolution and pixel format instruction may be furtherpassed to the video provider 555 which in turn may optimize the decodingpath. The video stream handler 540 connects a window 850 and a videoprovider. In use, the video provider 555 may be a video stream frommonitoring sensor 210 surveillance camera and the window 850 may be awindow object displayed on the monitor.

Layout module 520 may be further used to manipulate video stream handler540. When a user accesses a portion of a monitor 245 using a peripheraldevice (e.g. by clicking a mouse), the inputs are relayed to the videostream handler 540. Depending on the input, the video being displayingusing the video stream handler 540 may be paused, rewound, played back,or have any other operation performed thereon. Panning, tilting, orzooming functions can also be invoked through the video stream handler540 which preferably instructs the window 850 to pan tilt or zoom, oralternatively video stream handler can instruct the abstract videoprovider 555 to move the monitoring sensor 210 to monitor a particularobject or instance.

In certain embodiments of the monitoring system 100 the video streamhandler 540 can monitor the status of abstract video provider 555. Thus,video stream handler 540 may determine, for example, whether abstractvideo provider 555 still maintains a connection to its video source 213or whether the connection has been lost and needs to be reestablished.Layout module 520 periodically causes video stream handler 540 to updateitself and thereby can determine the connection status of the abstractvideo provider 555 and, if the abstract video provider 555 is connectedand has a new video frame available, can cause a video frame from theabstract video provider 555 to be transferred to and displayed in window850 and thereby on the identified monitor 245.

It should be noted that the above description of the display engine andvideo stream handling is purely exemplary and the invention should notbe limited to the components and methods described. Those skilled in theart will appreciate that there are multiple ways of utilizing variousmodules and hardware elements in order to achieve the desired objects ofreceiving, transferring, modifying, converting and/or outputting videostreams.

With reference to FIG. 5, in certain embodiments of monitoring system100, metadata stream is received along with an information stream fromvideo source 213. The acquisition of metadata from event coordinationmodule 320 is implemented in the metadata stream handler 545, abstractmetadata provider 560 and specific metadata provider 575 with metadatarenderer 530 finally creating a visual representation of the metadata.The abstract metadata provider 545 is contains the interface definitionsof functions such as connect, login, and any other relevant functionnecessary to access the metadata. The specific metadata provider 575 isan instance that contains the implementation specific to the source. Inuse, a specific metadata provider 575 that matches the specific sourcefrom which the metadata is received is created. The specific metadataprovider 575 is then manipulated, by the abstract metadata streamhandler 545 using the interface of the abstract metadata provider 560.These modules allow metadata that can be generated by a monitoringsensor 210 that uses a distinct format to be received by and interpretedby the monitoring system 100 and eventually displayed to the user.

In certain embodiments metadata received from video source 213 maycontain detailed information about the event monitored. The ability toreceive and interpret this metadata may allow for performance ofintelligent analytics and/or enhanced surveillance by a human user ofthe monitoring system 100. For example, metadata received from videosource 213 may be “alert, illegally parked car at (200, 120), (300,160)” in the video information stream being received from video source213. Upon receiving such metadata, monitoring system 100 may in oneembodiment, use proxy module 125, to convert the metadata to a syntaxneutral format. The syntax neutral format may include, for example, aglobally unique sensor identifier (provider URL) and a primitive definedas: primitive (1 . . . . N) Type (Box, Circle, Text, Polygon) Typespecific data (coordinates of the primitive) Color. For example, wherethe metadata is an alert of an illegally parked car having the form“alert, illegal parked car at (200, 120), (300),(160).” The syntaxneutral format may be:

URL: NetDVMS://demo:demo@192.168.7.72:800/[Parking Lot] Camera,Primitive(0)   Type: Text   Data: Illegally Parked Car   Color: RedPrimitive (1)   Type: Box   Data: (200, 120) (300, 160)   Color: Green,Time: 12:12:12:222 03/12/2008.

For every video frame, a new metadata packet may be created and sent toevent coordination module 320 which will in turn trigger metadatarenderer 530 which will utilize metadata stream handler 545, abstractmetadata provider 560, and specific metadata provider 575 as describedabove in order to create a visual representation of the metadata on themonitor of the user requesting the metadata. In the example used above,if a user of monitoring computer 215 was viewing the video stream fromvideo provider 213 that had the metadata described above, and the userrequested to view the metadata, the metadata renderer 530 would cause agreen box to be drawn around the illegally parked car and red textinforming the user that the car was an “Illegally Parked Car” is causedappear. This will notify the user of monitoring 215 about the illegallyparked car, and will focus the user's attention on the specific areawhere the car is located thereby. It should be noted that it will beapparent to those skilled in the art that metadata may be rendered inany number of ways, and that the method described above is exemplary andshould not be interpreted to be limiting.

With reference to FIG. 9, there is shown and described an embodiment ofa mapping graphical user interface (“GUI”) 1100 of a monitoring system100. This is an exemplary GUI that users of the monitoring system 100may use to select streams to monitor, receive alerts, send transferstreams to various other monitoring computers and place them graphicallyon a location map thereby informing the users of the location of thesensor 1120 that is producing the video stream. The interface preferablyincludes a map 1110 of the area being monitored, various camera icons1120 and auxiliary sensor icons 1140, alert notification area 1130, mapnavigation controls 1160, remote monitor controls 1170 and a remotevideo navigation panel 1180. Navigation panel 1180, may include a button(within the navigation controls 60, or elsewhere) that automaticallybrings the user to the location of the camera icon 1120 that representsthe camera producing the video stream the user is monitoring.

The map 1110 of the mapping GUI 1100 may be any type of map. It may begeographically accurate, such as one that may be found in a typicalatlas, or alternatively it may be a representative schematic or adiagram of the area being monitored. Additionally, the map 110 may betwo or three dimensional.

The map 1110 may be navigated, by a user, by using map navigationcontrols 60. The map navigation controls may allow the users to panaround the map and/or zoom in on areas of interest. The controls maytake on any form, for example icons such as arrows, and may beconfigured to respond to any instruction such as mouse clicks, atouchpad, and/or voice commands.

The map 1110 preferably includes camera icons 1120 and auxiliary icons40 imposed over the map, with the camera icons 1120 representing thelocation of various cameras throughout the area being monitored and theauxiliary sensor icons 1140 representing the location of variousmonitoring sensors throughout the area being monitored. The users of themapping GUI 1100 may interact with the various camera icons 1120auxiliary sensor icons 1140 and thereby access the information streamassociated with the camera 210 or monitoring sensor 210 beingrepresented by the respective icon. The interaction with the icons 1120,1140 may be caused in any way including, but not limited to, clickingthe mouse pointer over the icon, touching the icon via a touch-pad,referencing the icon via a voice command, or other indicative methods.

When a user selects a camera icon 1120, a video window object 1120 ispreferably displayed for the user on the monitor. The video windowobject 1120 preferably displays the stream of information received fromthe camera the icon of which was interacted with by the user. The windowobject 1120 may have permanent controls embedded in it, or the controlsmay fade in on or out (to preserve viewing area) due to user interactionthat may be, by way of non-limiting example, dragging a mouse cursorover a particular portion of the video window object. The controls ofthe window object may include any useful and compatible control, by wayof non-limiting examples the controls may include icons representingbuttons to pause, rewind, fast forward the stream and pan, tilt, and/orzoom over the stream and may include controls for additional features(described in further detail below) such as motion-detection,night-vision, etc. The panning, tilting and zooming over the stream maybe digital or optical. Thus, the user of the system may focus andenlarge particular areas of interest displayed in the window object1120.

The mapping GUI 1100 allows the user of the mapping GUI 1100 to selectusers to monitor various video streams. The mapping GUI 1100 may includemonitor icons 1170 representing various computer monitors 245 locatedthroughout the monitoring system 100. These monitors 245 may all beconnected to the same computer and may even be part of the terminalwhere the mapping GUI 1100 is being utilized, alternatively the monitorsmay also represent remote monitors connected to remote monitoringcomputers that are connected to the monitoring system 100 via thenetwork. In use, by way of non-limiting example, the user of the mappingGUI 1100 may drag, using a mouse, a video window 1120 showing a videostream from one of the cameras, to a monitor icon 1170 representing aremote monitor, thereby preferably giving the user of the remote monitorthe ability to view the video stream. The user of the remote computermay view the stream by having a window with the video stream pop up ontheir monitor, or by receiving a prompt to open a video window with thestream, though any other method may be utilized. The user viewing thevideo stream may access a control 1180 that reorients the map 1110 tothe location of the camera icon 1120 that represents the video cameraproducing the video stream.

The mapping GUI 1100 may further include a remote video navigation panel80, which preferably provides the user of the mapping GUI 1100 with theability to navigate various remote computer monitors throughout thesystem. The tools may any type of tool including scroll bars, shortcuts,links, etc.

The mapping GUI 1100 may further include an alert notification area 1130where alerts and alarms may appear. In use, as described above, theremay be situations where the various monitoring sensors may detect eventsthat were predetermined to set off alarms or alerts. These alarms, whichmay be messages, sounds, flashing window objects, or any other type ofindicator, may pop up on the mapping GUI 1100. The alerts and alarms maypop up on the GUI 1100 on the bottom right hand corner, or any otherpredetermined area, and notify the user of the event.

With reference to FIG. 10, there is shown and described an embodiment ofa live video review GUI 1200 of a monitoring system 100. This is anexemplary GUI that users of the monitoring system 100 may use to monitorspecific streams of live video from various monitoring sensors 210. Thelive review GUI 1200 may include one or more window objects 1210displaying live video streams, alert notification area 1130, and a videonavigation panel 20. This GUI allows users to simultaneously viewselected live video streams from specific monitoring sensor 210 of theirchoosing. The live video review GUI 1200 may further include an embeddedwebpage 1225, which may be used to access various websites on theinternet.

The video windows 1210 include a video information bar 1215, which maycontain any relevant information about the video stream, such as, forexample time, camera, or other indicative information. The video windowobjects 1210 may have the same functionality as the video window 1120 orany other functionality so long as video is viewable. In variousembodiments it is envisioned that a video navigation panel 1220 may beused to control the video stream displayed in window 1210. The videonavigation panel 1220 may be always visible, sometimes visible, or onlyvisible upon user request. The video navigation panel 1220 may includeany relevant tools, for example the ability to pause video, play video,pan, tilt, zoom, etc.

The live video review GUI 1200 may also include an event notificationarea 1230, similar to the alert notification area 1130, which may alertthe user of various occurrences in the system.

It should be noted that the arrangement of the objects on the varioususer interfaces described above and below is only exemplary. Any layoutmay be utilized with different icons, various sized windows, etc. Invarious embodiments of the invention it is envisioned that users maycustomize the layout to fit their preferences by, for example draggingand resizing various objects of the GUI, or in any other manner.

With reference to FIG. 11, there is shown and described an embodiment ofa video review GUI 1300 of a monitoring system 100 which may be accessedby a user of the monitoring system 100 that wants to review the recordedlive video stream. The video review GUI 1300 is similar to the livevideo review GUI 1200 except that it preferably includes an additionaltimeline control 1370, and individual time-line indicators 1340.

When reviewing recorded video, it is useful to know precisely when aparticular event occurred, therefore in the present embodiment atimeline indicator 1340, which indicates which time segment is beingdisplayed in the video window object 1210, is included.

The timeline control 1370 may be used to review video streams. Thetimeline may resemble a ruler that can be dragged in order to navigate.The timeline 1370 may be bookmarked by the operator in order to definetime segments of extra importance. The bookmarks 1355 and 1360 may beconfigured to contain additional information about the segment beingbookmarked. Furthermore, the timeline control 1370, may includeadditional navigation tools 1345, which may include any tool such asfast forward, play, or other navigational features.

With reference to FIG. 12, there is shown and described an embodiment ofa motion detection based search analysis that can be performed on astream of video information. Motion detection GUI 1201 may be accessedfrom the video review GUI 1300. In an embodiment of monitoring system100, motion is detected by performing a frame comparison. By performingframe comparison, monitoring system 100 can detect changes that occurover time in the area being monitored by monitoring sensor 210 and sendthose streams of video information to be displayed in GUI 1300.

Frame comparison may be performed by drill down module 390 by firstacquiring a reference frame (which may be selected by a user). Thisframe may be decoded from its compressed format (e.g. MJPEG or MPEG4) toan uncompressed 2-dimensional array of bytes. Each pixel of the frameconsists of 3 bytes, 1 byte for the red component of the pixel, 1 bytefor the green component of the pixel and 1 byte for the blue componentof the pixel. The value of the byte ranges from 0 to 255, where 0 is“off” and 255 is “full intensity.” By specifying different intensitiesfor the 3 components more than 16 million different hues/shades can bemade. The dimensions of the uncompressed frame matches the resolution ofthe reference frame to enable proper comparison. If the frame is 640 by480 pixels, the array will contain 640 columns and 480 rows of 3 bytevalues, each value specifying the pixel color.

Thereafter another frame is acquired from a video source. The distancein time between the reference frame and the new frame may be specifiedby the user. In some cases, a fine combing of the video data is needed(e.g. to focus on where an object the subject was holding was placed),and thus the distance in time may be very small, less than one second,while other times the distance in time can be larger than 10 seconds(e.g. to monitor when a certain vehicle left the parking lot).

Like the reference frame, the new frame is decompressed into a2-Dimensional array. Depending on the area being examined for motion,different values of arrays of reference frame and the new frame areextracted and compared to one another. The difference between each ofthe components of each frame is then examined (e.g. Dred=Abs(PixelR.Red−PixelN.Red), Dgreen=Abs (PixelR.Green−PixelN.Green),Dblue=Abs (PixelR.Blue−PixelN.Blue) where Dred represents the absolutevalue of difference in color between the red component of the pixel ofthe reference frame and the red component of the pixel of the new frame,Dgreen represents the absolute value of difference in color between thegreen component of the pixel of the reference frame and the greencomponent of the pixel of the new frame, and Dblue represents theabsolute value of difference in color between the blue component of thepixel of the reference frame and the blue component of the pixel of thenew frame). If the difference in between one of the three componentsexceeds the sensitivity variable specified by the user then the systemwill consider the pixel to have changed value.

Once all the pixels have been compared between the two frames, thesystem determines whether sufficient number of pixels have changed towarrant a finding of a change between two frames. The number of pixelsthat need to have had changed may be set by a user of the monitoringsystem 100. Typically, the larger the number of pixels that need to havehad changed, the more dramatic the change. A user looking for a largechange would set the threshold higher than someone looking for a smallchange, thereby expediting the process by avoiding the need to lookthrough extra video

As shown and described in FIG. 12, in use, a user defines an area ofinterest 1515 in the stream of video information 1510. Thereafter, usingcontrols described in detail below the user can cause the drill downmodule 390 of monitoring system 100 to perform frame comparison. As thedrill down module 390 detects change between frames, the location of thechange may be initially indicated by displaying a red pixel 1204 insteadof the real pixel color. This change in pixel color alerts the user tofocus on the area of interest where certain events have been detected.If the user pauses the search, the red pixel gradually fades and becomestransparent in order to reveal the original pixel color. In oneembodiment, this creates a visible fading red trail showing a motion ofan object.

It should be noted that the frame comparison process described above ispurely exemplary and not limiting. It will be apparent to those skilledin the art that frame comparison may be performed in any of a multitudeof ways for example by using sections of pixels, using algorithmsinstead of (or in addition to) user inputs, or through use of differentvideo formats, or any other modification. It should be further notedthat although frame comparison has been described with reference tomotion detection, it may be used to detect other events such as thesetting of the sun, changes in lighting, appearance of new objects, orany other event that can be seen by a difference between frames.

As shown and described with reference to FIG. 13, a user performingframe comparison may be provided with a graphical user interfaceincluding controls 1301 to select pixel count threshold 910, pixelchange sensitivity 916, time step adjustment 920, and motion buttons 930to navigate within the displayed video. These buttons are exemplary andother controls may be utilized to give user the ability to set theproperties used in performing frame comparison.

In an embodiment, the monitoring system 100 can facilitate and expeditethe process of locating a point in time when some particular eventoccurred. For that purpose, in response to user input to video reviewGUI 1300 (e.g. clicking on an icon on timeline control 1370, afterselecting point in time of the recorded stream), monitoring system 100may generate a thumbnail review GUI 1500 (as shown and described withreference to FIGS. 15A and 15B) which preferably contains a windows 1505showing images that are temporally displaced within a recorded stream ofvideo information which itself is displayed in a larger separate window1510. The time-displaced windows 1505 can be “thumbnail” size. Thesewindows 1505 may contain static images, although in alternateembodiments they may be capable of video playback. Each of theseindividual frames 1505 may show an entire image that was recorded, or itmay be a section of that recorded image.

To attain this functionality, the monitoring system 100 may include aplurality of system modules 110 loaded on the hardware (described abovewith reference to FIG. 2). As shown and described with reference to FIG.14, system modules 110 of the monitoring system 100 preferably furtherinclude a drill down module 390 and a thumbnail review GUI control 1415which work together with the display engine 515, provider URL module525, layout module 520, and video stream from video source 213 todisplay the relevant video in a user-friendly manner for the user. Thethumbnail review GUI control 1415 provides the functionality ofthumbnail review GUI 1500, and the drill down module 390 provides thefunctionality of controls that can allow the operator to for exampledigitally pan to, tilt to, or zoom in on to focus the plurality of videoframes on a specific region of interest within the entire regioncaptured by a camera.

As shown and described with reference to FIGS. 15A and 15B, in use,after a request from a user, via thumbnail review GUI 1500 module, drilldown module 390 is passed a master video stream which may contain visualinformation from, for example a live feed from a camera on the premises,a live feed from a camera located off the premises, or a previouslyrecorded video stream from any storage medium and location. Drill downmodule 390 may then create a provider URL 525 and cause the provider URL525 to connect to the same video stream that was sent to the drill downmodule 390. The video stream may be identified by a unique resourceidentifier such as a URL. After URL 525, makes the connection to thevideo stream, layout manager 520 may display the master video stream inwindow 1510 that is rendered by display engine 515.

Drill down module 390 may also cause display engine 515 to create windowobjects 1505. For example display engine 515 may create twenty windowobjects 1505 with a 256 by 256 RBGA texture map attached. The drill downmodule 390 may keep track of the windows by keeping the references tothe window objects 1505, in a data structure, such as an array. It willbe apparent to one skilled in the art that any number of windows of anysize, in any configuration, with any type of texture map, may be used.

The window objects may be populated with image frames from video streamby provider URL 525. Provider URL 525 preferably chooses frames, locatedat specific time intervals away from the time of the video stream thatmay be selected in any way, for example through the use of a timelinecontrol 1170. The provider URL 525 preferably copies one image frameinto each window object 1505. This process may be performed in a loopfashion, with the provider URL 525 choosing the first image frame,placing it into a window object, then choosing a second image frame thatis a certain time interval removed from a timestamp of the first imageframe, and placing the second image frame into a second window object,etc. until all of the window objects are populated with images.Alternatively, all the window objects 1505 may be populated at once, orin sections.

It should be noted that in various embodiments the drill down manager390 may be utilized together with a multitude of modules in order tocreate window objects 1505 based upon any number of conditions. Forexample, in certain embodiments, the drill down manager 390 may be usedin combination with frame comparison, thereby the various window objects1505 may be generated based upon an occurrence of a change between theframes of the video stream. In further embodiments, drill down manager390 may be combined with analytics performed by video source 213,monitoring sensor 210, or event coordinator module 320 and the variouswindow objects 1505 may be created based upon occurrence of events orcertain conditions being satisfied.

In order to fit the image frame into a window object 1505, the displayengine 515 may convert and scale the image frame. The conversion andscaling preferably causes the image frame to match the type, size, andtexture map of the window objects that is configured to be thumbnailsize. For example, the image frame chosen from the video source may beconverted to RGBA and scaled to a 256 by 256 matrix using nearestneighbor interpolation, though any other method, including but notlimited to bilinear interpolation may be used. The scaled RGBA matrixmay then be copied into the window object texture map. The coordinatesof the window object may be adjusted so that the aspect ratio of thewindow object is the same as the aspect ratio of the master video imageframe. This way, the final image displayed in the window object willappear to be correctly scaled, even though the aspect ratio of thetexture map is 1 to 1. It should be noted that scaling and conversionmay be performed in any way so long as the scaled and converted imageframe is capable of being displayed in the window object.

The monitoring system 100 preferably enables the user to focus, bydigitally tilting and panning, on a specific area of the entire videosource, and having the window objects 1505 digitally zoom in on thatspecific area. A thumbnail review GUI 1415 should permit a user toselect a region of interest 1515 from the window 1510 displaying thevideo source. For example, this may be performed by a user drawing arectangle around the region of interest 1515 in the thumbnail review GUI1415. Many alternate methods of selecting a region of interest 1515 arepossible, for example placing a pre-sized object over the area ofinterest, and the invention should not be limited to any specificmethod.

After the region of interest 1515 is selected, it is passed to the drilldown module 390 which may then perform offset removal, normalization,aspect correction, and/or frame edge adjustment before having thedisplay engine 515 place the region of interest 1515 as the frame imageinside the window object 1510.

As shown in FIG. 16, depending on the programming language used and thedisplay medium employed, offset removal may be performed prior toplacing the frame image into the window object. Drill down module 390may use the coordinates of the display medium, the coordinates of thestream of information, and the coordinates of the region of interest1515 in order to derive the location of the region of interest 1515within the stream of information.

As shown in FIG. 17, the region of interest 1515 is preferablynormalized before being displayed on monitor 245. Normalization may benecessary because in current systems the position of windows on themonitor is generally specified in either pixel coordinates or in3-Dimensional space coordinates. The texture map, which, in currentsystems, constitutes the actual content of a window on the monitor mayhave dimensions that are different from the window itself. In order tospecify what portion of the texture map is displayed inside the window,normalized coordinates may be used. By way of non-limiting example, thecoordinate (0,0) may be used to represent the upper left corner of thetexture map, while (1,1) may represent the lower right corner. Thus, toshow the entire contents of the texture map in the window, thecoordinates (0,0), (1,1) are used, and may be manipulated in order toallow a user to zoom in on an area. By way of non-limiting example(0,0), (0.5, 0.5) may be used to show the upper left of the textureinside the window. Since the position of the window displaying thetexture map of an image on the monitor hasn't changed, it will appear tothe user as if the window zoomed in on a portion of the image. It ispossible that in alternate embodiments normalization may be accomplishedin an alternative ways. If computers and displays utilize operatingsystems that map texture maps in ways other than pixel or 3-Dimensionalspace coordinates, normalization may not be necessary, or may need to beimplemented in ways that are compatible with those systems.

As shown in FIG. 18, there may be instances where the aspect ratio ofthe window object does not match the aspect ratio of the region ofinterest 1515. This may occur in instances where the window object isset to a certain aspect ratio 1505, while the aspect ratio of the regionof interest 1515 is determined by the user. In such monitoring system100 configurations, the drill down module 390 may adjust the size of theregion of interest 1515 to make its aspect ratio match the aspect ratioof the window object 1505. In order to achieve this without stretchingor shrinking the subjects within the region of interest 1515, additionalarea of the master video stream may be automatically added, by the drilldown module 390, to the region of interest 1515.

As shown in FIG. 19, aspect ratio adjustment may cause the post ratioadjustment region of interest 1515 to lie outside of the master videostream. To prevent this, the drill down module 390 may automaticallyadjust the edges of the region of interest 1515 such that they liewithin the borders of the master video stream. This is done by eithersubtraction or addition of values to the coordinates of the region ofinterest 1515.

In one embodiment a user of monitoring system 100 can analyze certainportions of a video stream without viewing or scrolling through theentire video stream. Alternatively, a user of monitoring system 100 canview portions of multiple video streams that all correspond to the sameexact time period.

The window objects, generated by the monitoring system 100, may becapable of displaying video. The operator may provide any type ofapplicable input to the monitoring system 100, for example pressing akey on a keyboard, touching a touchpad, or clicking a specific button ormenu on the monitor, instructing the monitoring system 100 to play thevideo, in each of the window objects, from the point in time displayedof the image displayed in each window object. It should be appreciatedthat the monitoring system 100 may be configured to have all of thethumbnails play video at once, alternatively some thumbnails may playvideo while others remain paused. Additionally, each window object mayhave controls for performing operations such as fast forward, rewind,pause, on the video in each individual window object. Enhancements, suchas motion tracking described above with reference to FIG. 12 may beshown within the thumbnails. In embodiments where the individual windows1505 contain static images, clicking on such windows 1505 preferablyresults in the video playing back in source window 1510 or anotherlocation, starting from the point in time of the static image in awindow 1505.

The GUI of the monitoring system 100 is purely exemplary, and thelocation and size and shape of the window objects, buttons, etc. maytake on any configuration. Additionally, in various embodiments of themonitoring system 100, the user may have the ability to alter thepositions and sizes of the various window objects on the monitor.

It should be noted that while the various GUIs are shown herein as beingdisplayed on a single monitor, it should be appreciated that any numberof monitors may be utilized. By way of non-limiting example, mapping GUI1100 may be displayed on a video wall of multiple monitors with onemonitor showing the map 1110, and the other monitors showing variouslive video window objects 1120. These window objects can, for example,be particular portions of a longer video being displayed on GUI 1100, ormay be unrelated video monitors.

It should be noted that although the embodiments described herein referto multiple software modules for performing the various functions ofmonitoring system 100, other embodiments could be implemented with asingle module incorporating the functions of several, or all, of themodules. The precise design of the software and the programming languageused may be designed differently within the scope of the presentinvention. The software modules can be created using art recognizedprogramming languages, including but not limited to C, C++, Java, C#, orPHP or any combination of known or hereafter developed programminglanguages that allow the functionality described.

Those skilled in the art will recognize that the method and system ofthe present invention has many applications, may be implemented in manymanners and, as such is not to be limited by the foregoing embodimentsand examples. In other words, functional elements being performed by asingle or multiple components, in various combinations of hardware andsoftware, and individual functions can be distributed among softwareapplications. In this regard, any number of the features of thedifferent embodiments described herein may be combined into one singleembodiment and alternate embodiments having fewer than or more than allof the features herein described are possible. Functionality may alsobe, in whole or in part, distributed among multiple components, inmanners now known or to become known. Thus, myriadsoftware/hardware/firmware combinations are possible in achieving thefunctions, features, interfaces and preferences described herein.Moreover, the scope of the present invention covers conventionally knownand features of those variations and modifications through the systemcomponent described herein as would be understood by those skilled inthe art. It is the intention, therefore, to be limited only as indicatedby the scope of the claims appended hereto.

It is also to be understood that the following claims are intended tocover all of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall therebetween.

1. A computer implemented method of performing video surveillance on acomputer system, wherein the computer system comprises a video and adisplay with a graphical user interface, the graphical user interfaceincluding a source window to view play back of the video, thesurveillance method comprising: determining an area of interest of aframe of the video based upon a selection of the area of interest withinthe source window; and generating a plurality of windows within thegraphical user interface, each of the windows displaying the area ofinterest of the frame of the video at a point in time, wherein thegraphical user interface includes controls to manipulate the playback ofthe video from a point in time relative to each displayed portion. 2.The method of claim 1, wherein the step of generating includes the stepof determining the point in time of at least one of the windows.
 3. Themethod of claim 2, wherein the step of determining the point in time ofat least one of the windows is based upon an input by a human operator.4. The method of claim 3, wherein the windows generated by thegenerating step are representative of portions of the video displacedfrom one another by a pre-determined period of time.
 5. The method ofclaim 4, wherein the pre-determined period of time is defined by a humanoperator.
 6. The method of claim 2, wherein the step of determining thepoint in time of at least one of the windows is based upon a detectionof movement in the video.
 7. The method of claim 2, wherein the step ofdetermining the point in time of at least one of the windows is basedupon a detection of an event in the video.
 8. The method of claim 1,wherein the selection of an area of interest is performed by drawing agraphical box around the area of interest on the source window.
 9. Themethod of claim 1, wherein determining an area of interest is performedbased upon a detection of movement in an area of the video.
 10. Themethod of claim 1 wherein determining an area of interest is performedbased upon a detection of an event in an area of the video.
 11. Themethod of claim 1, wherein the controls include zoom controls.
 12. Themethod of claim 1, wherein the windows are thumbnail sized.
 13. Themethod of claim 1, wherein the size of each window can be altered.
 14. Acomputer implemented method of performing video surveillance on acomputer system, wherein the computer system comprises a video and adisplay with a graphical user interface, the graphical user interfacecomprising a map designating a location of at least one monitoringsensor and a source window to view play back of a video, thesurveillance method comprising: displaying a video being detected by theat least one monitoring sensor; displaying a section of the maprepresenting the location of the at least one monitoring sensordetecting the video.
 15. The method of claim 14, wherein the videodisplayed is displaced based upon an instruction received through anetwork.
 16. The method of claim 14, wherein the instruction is issuedby a human operator. 17.-21. (canceled)
 22. An improved computerimplemented graphical user interface for monitoring video surveillancecomprising: a viewing region for viewing a video; a video navigationregion for manipulating the viewing of the video; a thumbnail regionwhich dynamically displays select frames of the video based on apre-defined criteria each frame having a navigation region for viewingof the video beginning with the displayed frame; wherein selectingindividual frames of the video populates the viewing region with saidindividual frame and plays the video beginning with the selected frame.23. An improved graphical user interface of claim 22 further comprisingan alert notification region for alerting a user of the graphical userinterface to a pre-defined occurrence in the video.